Clinical diagnosis and treatment planning system and methods of use

ABSTRACT

A spinal disorder diagnosis and treatment planning system is provided. The diagnosis and treatment planning system includes a mixed reality holographic display including at least one processor, at least one camera, at least one sensor, and being configured to acquire data points corresponding to a surface of a body adjacent to vertebral tissue. A computer database is configured to transmit imaging of the body including the vertebral tissue to the mixed reality holographic display. The mixed reality holographic display is configured to display a first holographic image of the vertebral tissue superimposed with a body image including the surface. Methods are also disclosed.

TECHNICAL FIELD

The present disclosure generally relates to medical systems for thediagnosis and treatment of musculoskeletal disorders, and moreparticularly to a system and method for spine disorder diagnosis andtreatment planning.

BACKGROUND

Spinal disorders such as degenerative disc disease, disc herniation,osteoporosis, spondylolisthesis, stenosis, scoliosis and other curvatureabnormalities, kyphosis, tumor and fracture may result from factorsincluding trauma, disease and degenerative conditions caused by injuryand aging. Spinal disorders typically result in symptoms including pain,nerve damage, and partial or complete loss of mobility.

Clinical evaluation and diagnosis of spinal disorders can be based onphysical examination and medical imaging of a patient. Physicalexamination may include evaluation of physical limitations in range ofmotion, evidence of instability, observable deformities and patient painresponses. Such physical examination and medical imaging is employed toformulate clinical diagnosis and treatment planning for the patient.Treatment planning may include non-surgical treatments and/or surgicaltreatments of spinal disorders. Non-surgical treatment can includemedication, rehabilitation and exercise, which can be effective,however, may fail to relieve the symptoms associated with thesedisorders. Surgical treatment of these spinal disorders includescorrection, fusion, fixation, discectomy, laminectomy and implantableprosthetics. As part of these surgical treatments, interbody devices canbe employed with spinal constructs, which include implants such as bonefasteners and vertebral rods to provide stability to a treated region.This disclosure describes an improvement over these prior technologies.

SUMMARY

In one embodiment, a spinal disorder diagnosis and treatment planningsystem is provided. The diagnosis and treatment planning system includesa mixed reality holographic display including at least one processor, atleast one camera and at least one sensor. The mixed reality holographicdisplay is configured to acquire data points corresponding to a surfaceof a body adjacent to vertebral tissue. A computer database isconfigured to transmit imaging of the body including the vertebraltissue to the mixed reality holographic display. The mixed realityholographic display is configured to display a first holographic imageof the vertebral tissue superimposed with a body image including thesurface. In some embodiments, methods are disclosed.

In one embodiment, the spinal disorder diagnosis and treatment planningsystem comprises a tangible storage device including computer-readableinstructions. A mixed reality holographic headset includes a centralprocessor and a holographic processor, and one or more cameras andsensors. One or more processors execute the instructions in operation ofthe system for: imaging a body including vertebral tissue in anon-surgical environment; scanning in real-time a surface of the bodyadjacent to the vertebral tissue with the mixed reality holographicheadset; registering a first holographic image of the vertebral tissuewith a body image of the scanned surface in a common coordinate system;and displaying in real-time the first holographic image and the bodyimage in the common coordinate system with the mixed reality holographicheadset.

In one embodiment, the spinal disorder diagnosis and treatment planningsystem comprises a tangible storage device comprising computer-readableinstructions. A mixed reality holographic headset includes a centralprocessor and a holographic processor, and one or more cameras andsensors. One or more processors, execute the instructions in operationof the system for: imaging a body including vertebral tissue in anon-surgical environment; scanning in real-time a surface of the bodyadjacent to the vertebral tissue with a mixed reality holographicheadset; determining a surgical treatment configuration for thevertebral tissue; registering a first holographic image of the vertebraltissue and/or a second holographic image of the surgical treatmentconfiguration with a body image of the scanned surface in a commoncoordinate system; and displaying in real-time the first holographicimage and/or the second holographic image with the body image in thecommon coordinate system with the mixed reality holographic headset.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will become more readily apparent from thespecific description accompanied by the following drawings, in which:

FIG. 1 is a perspective view of components of one embodiment of adiagnosis and treatment planning system in accordance with theprinciples of the present disclosure;

FIG. 2 is a plan view of components of one embodiment of a diagnosis andtreatment planning system in accordance with the principles of thepresent disclosure;

FIG. 3 is a perspective view of components of one embodiment of adiagnosis and treatment planning system including a representation ofimaging of vertebrae in accordance with the principles of the presentdisclosure;

FIG. 4 is a schematic diagram illustrating components of one embodimentof a diagnosis and treatment planning system including a representationof imaging and steps of one or more embodiments of a method inaccordance with the principles of the present disclosure;

FIG. 5 is a flow diagram illustrating representative steps of one ormore embodiments of a method and a diagnosis and treatment planningsystem in accordance with the principles of the present disclosure;

FIG. 6 is a flow diagram illustrating representative steps of one ormore embodiments of a method and a diagnosis and treatment planningsystem in accordance with the principles of the present disclosure;

FIG. 7 is a schematic diagram illustrating components of one embodimentof a diagnosis and treatment planning system including a representationof imaging and steps of one or more embodiments of a method inaccordance with the principles of the present disclosure;

FIG. 8 is a schematic diagram illustrating components of one embodimentof a diagnosis and treatment planning system including a representationof imaging in accordance with the principles of the present disclosure;

FIG. 9 is a schematic diagram illustrating components of one embodimentof a diagnosis and treatment planning system including a representationof imaging in accordance with the principles of the present disclosure;

FIG. 10 is a schematic diagram illustrating components of one embodimentof a diagnosis and treatment planning system including a representationof imaging and steps of one or more embodiments of a method inaccordance with the principles of the present disclosure;

FIG. 11 is a flow diagram illustrating representative steps of one ormore embodiments of a method and a diagnosis and treatment planningsystem in accordance with the principles of the present disclosure;

FIG. 12 is a flow diagram illustrating representative steps of one ormore embodiments of a method and a diagnosis and treatment planningsystem in accordance with the principles of the present disclosure; and

FIG. 13 is a flow diagram illustrating representative steps of one ormore embodiments of a method and a diagnosis and treatment planningsystem in accordance with the principles of the present disclosure.

DETAILED DESCRIPTION

The exemplary embodiments of a spinal disorder diagnosis and treatmentplanning system are discussed in terms of medical devices for thetreatment of musculoskeletal disorders and more particularly, in termsof a system and method for spine disorder diagnosis and treatmentplanning. In some embodiments, the present diagnosing and treatmentplanning system includes a mixed reality holographic display or anaugmented reality holographic display, and is employed with a method forspinal disorder diagnosis and treatment planning, includingsuperimposing a holographic image of a patient's vertebral tissue with abody image including the surface of the patient and correlating theimages for real-time visualization of spine orientation and alignment.

In some embodiments, the present surgical system comprises a displayincluding a holographic display device. In some embodiments, the systemsand methods of the present disclosure comprise a mixed reality displayor an augmented reality display employed with diagnosis and treatmentplanning, as described herein, for example, for a cervical, thoracic,lumbar and/or sacral region of a spine.

In some embodiments, the present diagnosis and treatment planning systemand method includes imaging of a patient's vertebrae, for example,through two-dimensional (2D) imaging generated from radiographyincluding, for example, an X-ray or a bi-plane X-ray long film. In someembodiments, the imaging is generated during patient movement,including, for example, flexation and/or extension. In some embodiments,a computer converts the imaging to digital data and transfers thedigital data to a mixed reality headset, for example, a holographicheadset. In some embodiments, the computer utilizes software todetermine a surgical treatment configuration for the vertebral tissuethrough segmentation and/or three dimensional (3D) reconstruction of thevertebrae. In some embodiments, an image of the surgical treatmentconfiguration is transmitted to the headset for display from theheadset. In some embodiments, the image of the vertebrae with the bodyimage and/or the surgical treatment configuration is holographicallyoverlaid onto the actual patient, including, for example, a surface ofthe body of the patient. In some embodiments, the holographic overlaysare implemented to view vertebral positioning, including for example,orientation and alignment on the patient. In some embodiments, theheadset includes automated image processing for measurement andalignment of the vertebrae of the patient.

In some embodiments, the headset includes cameras, for example, one ormore depth sensing cameras. In some embodiments, the one or more depthsensing cameras are configured to spatially map surfaces of the patientin an environment for localizing and displaying content. In someembodiments, recognition markers are positioned on objects, for example,a back surface of the patient that can be recognized by the cameras onthe headset for displaying the content. In some embodiments, the contentdisplayed is a map of the back of the patient. In some embodiments, theone or more depth sensing cameras provide a real-time update of a 3Dvertebral model during a physical exam of the patient.

In some embodiments, the present diagnosis and treatment planning systemincludes a holographic display system that is implemented during aninitial evaluation, including, for example, a physical examination for asurgical procedure such that the image of the vertebrae with the bodyimage and/or the surgical treatment configuration is holographicallyoverlaid onto the actual patient to dynamically visualize patientanatomy for optimizing diagnosis and treatment planning. In someembodiments, the images are integrated with the patient through aholographic overlay. In some embodiments, the holographic overlayincludes images of the vertebrae with the body image and/or the surgicaltreatment configuration that is patient specific. In some embodiments,the image of the vertebrae with the body image and/or the surgicaltreatment configuration utilizes patient specific anatomy data generatedfrom images, for example, radiography, including, for example, an X-rayor a bi-plane X-ray long film. In some embodiments, the holographicoverlay is superimposed on a surface of the patient in the clinic priorto a surgical procedure for clinical diagnosis and treatment planning.In some embodiments, a surgical plan can be generated from images of thevertebrae with the body image and/or the surgical treatmentconfiguration that is uploaded into a navigation and/or robotic system,including for example, Mazor X™ and/or Stealthstation™ sold by MedtronicNavigation, Inc. having a place of business in Louisville, Colo. when asurgical procedure is required.

In some embodiments, the present diagnosis and treatment planning systemincludes recognition markers positioned relative to the patient to mapthe surface of the patient. In some embodiments, a scanner isimplemented to map the surface of the patient. In some embodiments, theholographic overlay is implemented in conjunction with depth sensingcameras and/or sensors for real-time visualization of vertebralorientation and alignment.

In some embodiments, the present diagnosis and treatment planning systemand methods include spatially located 3D holograms, for example,holographic overlays for displaying image guidance information. In someembodiments, the present diagnosis and treatment planning system andmethods include cameras, for example, depth sensing cameras. In someembodiments, the depth sensing cameras include infrared, laser, and/orred/green/blue (RGB) cameras. In some embodiments, depth sensing camerasalong with simultaneous localization and mapping are employed todigitize the patient, spinal anatomy, and/or the clinic room forspatially locating holograms and then displaying the digital informationholographically onto the patient and correlation of the image data withcurrent patient anatomy for real-time visualization of spinal alignment.In some embodiments, the present diagnosis and treatment planning systemand methods include software algorithms, for example, object recognitionsoftware algorithms that are implemented for spatially locating theholograms and for displaying digital information relative to patientanatomy. In some embodiments, machine learning algorithms are employedthat identify patient anatomical structures in patient images for imagesegmentation and 3D model generation as well as to identify patientexternal anatomy for spatial placement of holographic images. In someembodiments, machine learning algorithms generate automated measurementsof spinal alignment parameters in patient images, for example, includinga cobb angle, pelvic tilt, pelvic incidence, sacral slope, lumbarlordosis, thoracic kyphosis, cervical lordosis, and/or sagittal verticalaxis. In some embodiments, software algorithms are implemented in 3Dimage processing software employed for the surgical treatmentconfiguration including for example, software algorithms for importing,thresholding, masking, segmentation, cropping, clipping, panning,zooming, rotating, measuring and/or registering.

In some embodiments, the present diagnosis and treatment planning systemand methods include depth sensing cameras, for example, infrared, laser,and/or RGB cameras; spatial transducers, for example, electromagnetic,low energy Bluetooth®, and/or inertial measurement units; opticalmarkers, for example, reflective spheres, QR codes/patterns, and/orfiducials; and/or object recognition software algorithms to track aspatial position of a patient, for example, a patient's vertebral bodiesand update a digital representation in real time. In some embodiments,the present diagnosis and treatment planning system and methods include3D imaging software algorithms implemented to render and display changesin an anatomical position in real-time. In some embodiments, the presentdiagnosis and treatment planning system and methods include holographicdisplay technology, for example, optical waveguides to displayholograms, image guidance, and/or other digital information inreal-time.

In some embodiments, the present diagnosis and treatment planning systemand methods include labeling, for example, annotating of 3D data and/ormedical imaging of the patient's anatomy in real-time during a physicalexamination and/or in real-time, for example, to identify areas ofinterest and/or locations where a patient is experiencing pain. In someembodiments, during a physical examination and/or imaging while thepatient is moving, for example, flexing and/or extending, a medicalpractitioner has the ability to label an area of interest and/orlocations where a patient is experiencing pain. In some embodiments, thelabel assists the medical practitioner during a surgical procedure.

In some embodiments, the present diagnosis and treatment planning systemand methods include a software program for processing of 2D images forexample, radiography, including, for example, bi-plane X-ray long filmimages to reconstruct these images into 3D anatomy such that thepractitioner has freedom to view the patient images from anyangle/orientation and see anatomy data accurately positioned on thepatient.

In some embodiments, the present diagnosis and treatment planning systemand methods include real-time mapping of the patient's back curvature toupdate imaging of the patient's vertebrae as the patient is examinedwith flexation, extension and/or lateral bending maneuvers. In someembodiments, during imaging, patient anatomy is flexed at an appropriatelocation and is tracked with the patient's position as the practitionerexamines the patient.

In some embodiments, the present diagnosis and treatment planning systemis employed with methods for spinal disorder diagnosis and treatmentplanning. In some embodiments, the present diagnosis and treatmentplanning system is employed with methods including the step of imaging abody including vertebral tissue. In some embodiments, imaging isgenerated through 2D imaging generated from radiography, including, forexample, an X-ray or a bi-plane X-ray long film. In some embodiments,the present diagnosis and treatment planning system is employed withmethods including the step of acquiring data points corresponding to asurface of the body adjacent to the vertebral tissue with a mixedreality holographic display. In some embodiments, the present diagnosisand treatment planning system is employed with methods including thestep of transmitting the imaging to a computer database. In someembodiments, a computer utilizes software to determine a surgicaltreatment configuration for the vertebral tissue through segmentationand/or 3D reconstruction of the vertebrae. In some embodiments, thepresent diagnosis and treatment planning system is employed with methodsincluding the step of superimposing a holographic image of the vertebraltissue with a body image including the surface. In some embodiments, thepresent diagnosis and treatment planning system is employed with methodsincluding the step of displaying the holographic image and the bodyimage with the mixed reality holographic display. In some embodiments,the mixed reality holographic display system includes a processor,cameras, and sensors.

In some embodiments, the system of the present disclosure may beemployed for diagnosing and treatment planning for spinal disorders suchas, for example, degenerative disc disease, disc herniation,osteoporosis, spondylolisthesis, stenosis, scoliosis and other curvatureabnormalities, kyphosis, tumor and fractures. In some embodiments, thesystem of the present disclosure may be employed with other osteal andbone related applications, including those associated with diagnosticsand therapeutics. In some embodiments, the disclosed system may bealternatively employed in a diagnosing and treatment planning with apatient in a prone or supine position, and/or employ various approachesto the spine, including anterior, posterior, posterior mid-line, directlateral, postero-lateral, and/or antero-lateral approaches, and in otherbody regions. The system of the present disclosure may also bealternatively employed with procedures for treating the lumbar,cervical, thoracic, sacral and pelvic regions of a spinal column. Thesystem of the present disclosure may also be used on animals, bonemodels and other non-living substrates, such as, for example, intraining, testing and demonstration.

The system of the present disclosure may be understood more readily byreference to the following detailed description of the embodiments takenin connection with the accompanying drawing figures, which form a partof this disclosure. It is to be understood that this application is notlimited to the specific devices, methods, conditions or parametersdescribed and/or shown herein, and that the terminology used herein isfor the purpose of describing particular embodiments by way of exampleonly and is not intended to be limiting. In some embodiments, as used inthe specification and including the appended claims, the singular forms“a,” “an,” and “the” include the plural, and reference to a particularnumerical value includes at least that particular value, unless thecontext clearly dictates otherwise. Ranges may be expressed herein asfrom “about” or “approximately” one particular value and/or to “about”or “approximately” another particular value. When such a range isexpressed, another embodiment includes from the one particular valueand/or to the other particular value. Similarly, when values areexpressed as approximations, by use of the antecedent “about,” it willbe understood that the particular value forms another embodiment. It isalso understood that all spatial references, such as, for example,horizontal, vertical, top, upper, lower, bottom, left and right, are forillustrative purposes only and can be varied within the scope of thedisclosure. For example, the references “upper” and “lower” are relativeand used only in the context to the other, and are not necessarily“superior” and “inferior”.

As used in the specification and including the appended claims,“treating” or “treatment” of a disease or condition refers to performinga procedure that may include administering one or more drugs to apatient (human, normal or otherwise or other mammal), employingimplantable devices, and/or employing instruments that treat thedisease, such as, for example, microdiscectomy instruments used toremove portions bulging or herniated discs and/or bone spurs, in aneffort to alleviate signs or symptoms of the disease or condition.Alleviation can occur prior to signs or symptoms of the disease orcondition appearing, as well as after their appearance. Thus, treatingor treatment includes preventing or prevention of disease or undesirablecondition (e.g., preventing the disease from occurring in a patient, whomay be predisposed to the disease but has not yet been diagnosed ashaving it). In addition, treating or treatment does not require completealleviation of signs or symptoms, does not require a cure, andspecifically includes procedures that have only a marginal effect on thepatient. Treatment can include inhibiting the disease, e.g., arrestingits development, or relieving the disease, e.g., causing regression ofthe disease. For example, treatment can include reducing acute orchronic inflammation; alleviating pain and mitigating and inducingre-growth of new ligament, bone and other tissues; as an adjunct insurgery; and/or any repair procedure. Also, as used in the specificationand including the appended claims, the term “tissue” includes softtissue, ligaments, tendons, cartilage and/or bone unless specificallyreferred to otherwise.

The following discussion includes a description of a diagnosis andtreatment planning system including mixed and/or augmented realitytechnology, holographic overlays, related components and methods ofemploying the diagnosis and treatment planning system in accordance withthe principles of the present disclosure. Alternate embodiments are alsodisclosed. Reference is made in detail to the exemplary embodiments ofthe present disclosure, which are illustrated in the accompanyingfigures. Turning to FIGS. 1-10 , there are illustrated components of adiagnosis and treatment planning system 10.

The components of diagnosis and treatment planning system 10 can befabricated from biologically acceptable materials suitable for medicalapplications, including metals, synthetic polymers, ceramics and bonematerial and/or their composites. For example, the components ofdiagnosis and treatment planning system 10, individually orcollectively, can be fabricated from materials such as stainless steelalloys, aluminum, commercially pure titanium, titanium alloys, Grade 5titanium, super-elastic titanium alloys, cobalt-chrome alloys,superelastic metallic alloys (e.g., Nitinol, super elasto-plasticmetals, such as GUM METAL®), ceramics and composites thereof such ascalcium phosphate (e.g., SKELITE™), thermoplastics such aspolyaryletherketone (PAEK) including polyetheretherketone (PEEK),polyetherketoneketone (PEKK) and polyetherketone (PEK), carbon-PEEKcomposites, PEEK-BaSO₄ polymeric rubbers, polyethylene terephthalate(PET), fabric, silicone, polyurethane, silicone-polyurethane copolymers,polymeric rubbers, polyolefin rubbers, hydrogels, semi-rigid and rigidmaterials, elastomers, rubbers, thermoplastic elastomers, thermosetelastomers, elastomeric composites, rigid polymers includingpolyphenylene, polyamide, polyimide, polyetherimide, polyethylene,epoxy, bone material including autograft, allograft, xenograft ortransgenic cortical and/or corticocancellous bone, and tissue growth ordifferentiation factors, partially resorbable materials, such as, forexample, composites of metals and calcium-based ceramics, composites ofPEEK and calcium based ceramics, composites of PEEK with resorbablepolymers, totally resorbable materials, such as, for example, calciumbased ceramics such as calcium phosphate, tri-calcium phosphate (TCP),hydroxyapatite (HA)-TCP, calcium sulfate, or other resorbable polymerssuch as polyaetide, polyglycolide, polytyrosine carbonate,polycaroplaetohe and their combinations.

The components of diagnosis and treatment planning system 10,individually or collectively, may also be fabricated from aheterogeneous material such as a combination of two or more of theabove-described materials. The components of diagnosis and treatmentplanning system 10 may be monolithically formed, integrally connected orinclude fastening elements and/or instruments, as described herein.

Diagnosis and treatment planning system 10 can be employed, for example,before a minimally invasive procedure, including percutaneoustechniques, mini-open and open surgical techniques to manipulate tissue,deliver and introduce instrumentation and/or components of spinalconstructs at a surgical site within a body of a patient, for example, asection of a spine.

Diagnosis and treatment planning system 10 is implemented during aninitial evaluation, including, for example, during a physicalexamination for a surgical procedure such that imaging of patientvertebrae with a body image and/or a surgical treatment configurationcan be holographically overlaid onto the actual patient to dynamicallyvisualize patient anatomy for optimizing diagnosis and treatmentplanning. Diagnosis and treatment planning system 10 utilizes a mixedreality and/or augmented reality display, for example, toholographically overlay imaging of patient vertebrae with the body imageand/or the surgical treatment configuration specific to a patient onto asurface of the patient to assist in diagnosing and treatment planningfor the patient.

Diagnosis and treatment planning system 10 includes a mixed realityholographic display, for example, an optical see-through headset 12, asshown in FIG. 2 . Headset 12 is configured to acquire data pointscorresponding to a surface of a body of a patient adjacent to vertebraltissue. The data points include, for example, 3D mapping of the surfaceof the body. Headset 12 is configured to communicate with a database 14loaded on a computer 42 that transmits imaging 16 of the body includingthe vertebral tissue to headset 12. Headset 12 is configured to displaya holographic image 18 of the vertebral tissue superimposed with a bodyimage 20 including the surface of the body in real-time. Headset 12 isconfigured to dynamically display in real-time orientation and alignmentof the vertebral tissue in a non-surgical environment, for example, aclinic, medical practitioner office, examination room, hospital and/ormedical evaluation and diagnosis facility prior to surgery as describedherein.

Headset 12 is configured to display a surgical treatment configurationholographic image 23 superimposed with body image 20. Data pointscorresponding to the surface of the body of the patient adjacent tovertebral tissue are transmitted from headset 12 to database 14 suchthat database 14 can determine a surgical treatment configuration 22 forthe vertebral tissue. The surgical treatment configuration data is thentransferred to headset 12 for display as surgical treatmentconfiguration image 23. The surgical treatment configuration 22 includessegmentation of the vertebral tissue and/or a surgical reconstruction ofthe vertebral tissue, as described herein and show in FIG. 4 .

Headset 12 includes a processor 24, for example, a central processingunit (CPU). Processor 24 is configured to execute one or moreinstructions, for example, software instructions in operation of headset12, as described herein. Processor 24 functions as the primarycoordinating component of headset 12 and is configured to accessprograms, data, and/or other functions from random access memory (RAM)when called by an operating system (OS) of headset 12. Processor 24interprets instructions that are related to ordered tasks before sendingit back to the RAM for execution via a bus of headset 12 in the correctorder of execution.

Headset 12 includes a rendering processor, for example, a graphicsprocessor 25. Graphics processor 25 includes a graphics processing unit(GPU). Graphics processor 25 is configured to render images, animationsand/or video for display on headset 12. In some embodiments, processor24 instructs graphics processor 25 to render the images, animationsand/or video. Images rendered include, for example, image 18 of thevertebral tissue, body image 20 and/or surgical treatment configurationimage 23. Graphics processor 25 is configured to communicate with acamera 26 of headset 12 which captures a digital video image of the realworld and transfers the digital video image to graphics processor 25 inreal-time. Graphics processor 25 combines the video image feed withcomputer-generated images (e.g., virtual content), for example, image 18of the vertebral tissue, body image 20 and/or surgical treatmentconfiguration image 23 and displays the images on headset 12. In someembodiments, headset 12 alternatively or in addition to graphicsprocessor 25 includes a holographic processor 27. Holographic processor27, for example a holographic processing unit (HPU) is configured toconduct the processing that integrates digital video image data of thereal world, data for augmented reality and/or user input (see, forexample, the holographic processing unit sold by Microsoft Corporation,having a place of business in Redmond, Wash., USA).

Headset 12 includes camera 26, for example, a depth sensing camera.Camera 26 is disposed on a front side 29 of headset 12, as shown in FIG.2 . Camera 26 is configured to capture real-time digital video images ofthe patient, for example, the vertebral tissue and/or a map of a portionof the patient, including, for example, the patient's back for areal-time update of a 3D vertebral model during an examination, as shownin FIGS. 8 and 9 and/or real-time images of an external environment ofthe real world, for example, the clinic or examination room during anexamination. The real-time images captured by camera 26 are outputted toheadset 12 and displayed on a lens 30 of headset 12. The real-timeimages captured by camera 26, image 18 of the vertebral tissue, bodyimage 20 and/or surgical treatment configuration image 23 rendered fromgraphics processor 25 are displayed concurrently. In some embodiments,camera 26 includes an environment camera. In some embodiments, the depthsensing camera can work in tandem with the environment camera. In someembodiments, the depth sensing camera includes infrared, laser, and/orRGB cameras. In some embodiments, camera 26 includes a stereoscopiccamera, for example, a pair of cameras.

Headset 12 includes a sensor 28. Sensor 28 is disposed on front side 29of headset 12 and is a component of camera 26, as shown in FIG. 2 .Sensor 28 includes a 3D scanner 32 configured to determine and capturethe 3D mapping of the surface of the body so that, for example, areal-time update of a 3D vertebral model of the patient can bedisplayed, as shown in FIGS. 8 and 9 . 3D scanner 32 is configured todetermine and capture the 3D mapping of the surface of the body so thatimage 18 of the vertebral tissue, body image 20, surgical treatmentconfiguration image 23 and/or other images can be holographicallyoverlaid onto the patient through headset 12. In some embodiments,camera 26 along with simultaneous localization and mapping implementedby 3D scanner 32 digitizes the patient, spinal anatomy, and/or theclinic or examination room for spatially locating holograms and thendisplays the digital information via lens 30 of headset 12. Digitalvideo (e.g., stereoscopic video) combined with 3D mapping of the surfaceof the body determined by 3D scanner 32 and image 18 of the vertebraltissue, body image 20 and/or surgical treatment configuration image 23is combined by graphics processor 25 for display.

In some embodiments, 3D scanner 32 implements simultaneous localizationand mapping (SLAM) technology to determine 3D mapping of the surface ofthe body. SLAM technology simultaneously localizes (finds the locationof an object/sensor with reference to its surroundings) and maps thelayout and framework of the environment for headset 12. This can be doneusing a range of algorithms that simultaneously localize and map theobjects.

In some embodiments, 3D mapping of the surface of the body can bedetermined through the use of 3D scanner 32, camera 26 and recognitionmarkers (not shown) positioned relative to the patient and/or on asurface of the patient to map the surface of the patient. In someembodiments, the recognition markers may be attached to the patient toprovide anatomic landmarks of the patient during the 3D scanningprocess. The recognition markers, alone or in combination with othertracking devices, such as inertial measurement units (IMU), may beattached to 3D scanner 32, camera 26, and/or the surgeon (e.g. throughheadset 12).

In some embodiments, 3D mapping of the surface of the body can bedetermined through the use of 3D scanner 32, camera 26, and/or forexample, spatial transducers, for example, electromagnetic, low energyBluetooth®, and/or inertial measurement units; optical markers, forexample, reflective spheres, QR codes/patterns, and/or fiducials; and/orobject recognition software algorithms to track a spatial position of apatient, for example, a patient's vertebral tissue, for example,vertebral bodies and/or a back of the patient and update a digitalrepresentation in real time.

In some embodiments, headset 12 includes sensor 28, motion sensors,acoustic/audio sensors (where the audio is transmitted to speakers (notshown) on headset 12), laser rangefinders, and/or visual sensors. Insome embodiments, headset 12 includes sensor 28 and additional sensorsincluding accelerometers, magnetometers, and/or gyroscopes which measuremotion and direction in space of headset 12 and enables translationalmovement of headset 12 in an augmented environment.

3D mapping of the surface of the body and/or image 18 of the vertebraltissue is registered via processor 24 functioning as a registrationprocessor. In some embodiments, processor 24 registers 3D mapping of thesurface of the body and/or image 18 of the vertebral tissue and surgicaltreatment configuration image 23. In some embodiments, the registeredimages can be uploaded to a computer 42, as described herein, externalto headset 12. The registered 3D mapping of the surface of the bodyand/or image 18 of the vertebral tissue will be automatically blendedwith body image 20. The registered images can be displayed on headset 12and/or can be projected over the patient as a holographic overlay.

Lens 30 includes a screen that employs holographic display technology,for example, optical waveguides to display holograms, image guidance,and/or other digital information in real-time. In some embodiments,headset 12 via lens 30 displays a 360° view through the patient of image18 of the vertebral tissue, body image 20 and/or surgical treatmentconfiguration image 23. In some embodiments, headset 12 includes, forexample, goggles or glasses (see, for example, similar goggles orglasses of HoloLens® or HoloLens® 2 (Microsoft Corporation, Redmond,Wash., USA); or Magic Leap® (Magic Leap, Inc, Florida, USA) and/orDreamGlass® (Dreamworld, Calif., USA)).

In some embodiments, headset 12 employs holographic display technologywhere light particles (e.g., photons) bounce around in a light enginewithin the device. The light particles enter through two lenses 30 ofthe headset 12 where the light particles ricochet between layers ofblue, green and red glass before reaching the back of the surgeon'seyes. Holographic images form when the light is at a specific angle. Insome embodiments, headset 12 includes a contact lens and/or an eye loop.In some embodiments, headset 12 includes a handheld device including,for example, a tablet or a smartphone. In some embodiments, system 10includes projector technology including a display plate as analternative to headset 12 or in addition to headset 12.

Imaging 16 is generated by an imaging device 36, as shown in FIG. 3 .Imaging device 36 is configured to generate images of a selected portionof the patient's anatomy, for example, vertebral tissue. Imaging device36 is configured to generate 2D images. In some embodiments, imagingdevice 36 includes, for example, radiography, including, for example, anX-ray or a bi-plane X-ray long film. Imaging 16 is converted into imagedata to store within database 14. In some embodiments, imaging 16 isconverted into image data by a software program. In some embodiments,the data points of imaging 16 can be transmitted wirelessly or uploadedinto headset 12. In some embodiments, a software program is implementedfor processing imaging 16 (e.g., a 2D image, for example, the bi-planeX-ray long film image) to reconstruct imaging 16 into 3D anatomy suchthat the practitioner has freedom to view the patient images from anyangle/orientation and to see anatomy data accurately positioned on thepatient. In some embodiments, the software program can include EOSsoftware programs (see, for example, the software programs sold by EOSImaging, Inc. having a place of business in St. Paul, Minn.).

In some embodiments, real-time mapping of the patient's back curvatureto update imaging of the patient's vertebrae can occur as the patient isexamined with flexation, extension and/or lateral bending maneuvers. Insome embodiments, during imaging 16, patient anatomy is flexed at anappropriate location and is tracked with the patient's position as thepractitioner examines the patient.

In some embodiments, imaging device 36 is configured to generate 3Dimages. In some embodiments, imaging device includes a CT scan, an MRscan, ultrasound, positron emission tomography (PET), and/or C-armcone-beam computed tomography.

Database 14 is stored on a tangible storage device 38 that includescomputer-readable instructions. In some embodiments, storage device 38includes a hard drive of computer 42. In some embodiments, storagedevice 38 is an external hard drive unit. In some embodiments, storagedevice 38 includes a magnetic storage device, for example, a floppydiskette, magnetic strip, SuperDisk, tape cassette, or zip diskette; anoptical storage device, for example, a Blu-ray disc, CD-ROM disc, CD-ftCD-RW disc, DVD-R, DVD+R, DVD-RW, or DVD+RW disc; and/or flash memorydevices, for example, USB flash drive, jump drive, or thumb drive,CompactFlash (CF), M.2, memory card, MMC, NVMe, SDHC Card, SmartMediaCard, Sony Memory Stick, SD card, SSD or xD-Picture Card. In someembodiments, storage device 38 includes online storage, cloud storage,and/or network media storage. In some embodiments, headset 12 can accessdatabase 14/storage device 38 wirelessly.

As shown in FIG. 5 , processor 24 and/or a processor 44, for example, aCPU of computer 42 execute the instructions in operation of system 10.Processor 24 and/or processor 44 execute instructions for imaging 16 ina non-surgical environment, scanning in real-time a surface of the bodyadjacent to the vertebral tissue with headset 12; registering image 18of the vertebral tissue with body image 20 in a common coordinatesystem; and displaying in real-time image 18 of the vertebral tissue andbody image 20 in the common coordinate system with headset 12. In someembodiments, processor 24 and/or processor 44 further determine surgicaltreatment configuration 22 for the vertebral tissue.

As shown in FIG. 6 , processor 24 and/or processor 44 executeinstructions for imaging 16 in a non-surgical environment; scanning inreal-time a surface of the body adjacent to the vertebral tissue withheadset 12; determining surgical treatment configuration 22 for thevertebral tissue; registering image 18 of the vertebral tissue and/orsurgical treatment configuration image 23 with body image 20 of thescanned surface in a common coordinate system; and displaying inreal-time image 18 of the vertebral tissue and/or surgical treatmentconfiguration image 23 with body image 20 in the common coordinatesystem with headset 12.

Computer 42 generates surgical treatment configuration 22, as shown inFIG. 7 via a software program. In some embodiments, the software programincludes, for example, Mazor X™, Mazor X™ Align, and/or Stealthstation™sold by Medtronic Navigation, Inc. having a place of business inLouisville, Colo. In some embodiments, the software program is 3D imageprocessing software that includes software algorithms employed for theprocedure planning including for example, software algorithms forimporting, thresholding, masking, segmentation, cropping, clipping,panning, zooming, rotating, measuring and/or registering. The softwareprogram is preloaded onto computer 42, the surgical treatmentconfiguration 22 is generated by the software program, and the surgicaltreatment configuration 22 is uploaded onto headset 12 where graphicsprocessor 25 renders surgical treatment configuration image 23 so thatit is outputted from lens 30 for display. In some embodiments, thesoftware program is alternatively preloaded onto headset 12, surgicaltreatment configuration 22 is generated from the software and headset 12displays surgical treatment configuration image 23 from lens 30.

In some embodiments, headset 12 implements software algorithms, forexample, object recognition software algorithms that are implemented forspatially locating the holograms (e.g., image 18 of the vertebral tissueand/or surgical treatment configuration image 23) and for displayingdigital information relative to patient anatomy. In some embodiments,machine learning algorithms are employed that identify patientanatomical structures in patient images (e.g., imaging 16) for imagesegmentation and 3D model generation as well as to identify patientexternal anatomy for spatial placement of holographic images. In someembodiments, machine learning algorithms generate automated measurementsof spinal alignment parameters in patient images, for example, includinga cobb angle, pelvic tilt, pelvic incidence, sacral slope, lumbarlordosis, thoracic kyphosis, cervical lordosis, and/or sagittal verticalaxis.

In some embodiments, during a physical examination and/or imaging 16while the patient is moving, for example, flexing and/or extending, amedical practitioner has the ability to label, including, for example,annotating an area of interest and/or locations where a patient isexperiencing pain directly onto imaging 16. In some embodiments, thelabel assists the medical practitioner during a surgical procedure.

In assembly, operation and use, diagnosis and treatment planning system10, similar to the systems and methods described herein, is employedwith a method for spinal disorder diagnosis and treatment planning.Diagnosis and treatment planning system 10 may also be employed prior toa surgical procedure, such as, for example, discectomy, laminectomy,fusion, laminotomy, laminectomy, nerve root retraction, foramenotomy,facetectomy, decompression, spinal nucleus or disc replacement and bonegraft and implantable prosthetics including plates, rods, and boneengaging fasteners.

In one embodiment, diagnosis and treatment planning system 10, similarto the components of the systems and methods described herein, isemployed in connection with one or more spinal disorder diagnosis andtreatment plans. See, for example, the embodiments and disclosure ofsystems and methods for spinal disorder diagnosis and treatmentplanning, shown and described in commonly owned and assigned U.S. patentapplication Ser. No. 16/855,695 filed, Apr. 20, 2020, and published asU.S. Patent Application Publication No. 2021/0330250, on Oct. 28, 2021,the entire contents of which being incorporated herein by reference.

In some embodiments, system 10 includes a method 100 for spinal disorderdiagnosis and treatment planning, as shown in FIG. 11 . In a step 102, abody including vertebral tissue of a patient is imaged to generateimaging 16. The body is imaged via an imaging device 36. In someembodiments, imaging device 36 includes radiography, including, forexample, an X-ray or a bi-plane X-ray long film. In some embodiments,imaging includes imaging the body in a non-surgical environment, asdescribed herein.

In a step 104, data points are acquired corresponding to a surface ofthe body adjacent to the vertebral tissue with a mixed realityholographic display. The mixed reality display includes headset 12.Headset includes processor 24, camera 26 and sensor 28. Camera 26 is adepth sensing camera, as described herein. In some embodiments, themixed reality display includes a handheld device. In a step 106, imaging16 is transmitted to computer database 14. In some embodiments, imaging16 is converted to data points by a software program, as describedabove. Computer database 14 is located on computer 42. In a step 108,holographic image 18 of the vertebral tissue is superimposed with bodyimage 20 including the surface of the patient, as described herein. In astep 110, holographic image 18 of the vertebral tissue and body image 20are displayed on headset 12. In some embodiments, holographic image 18of the vertebral tissue superimposed with body image 20 is displayed inreal-time on headset 12. In some embodiments, step 110 further includesdynamically displaying in real-time the orientation and alignment of thevertebral tissue in a non-surgical environment.

In an optional step 112, surgical treatment configuration 22 for thevertebral tissue is determined. In some embodiments, surgical treatmentconfiguration 22 includes a segmentation of the vertebral tissue and/ora surgical reconstruction of the vertebral tissue. In some embodiments,surgical treatment configuration 22 is determined and/or generated froma software program, as disclosed above, including, for example, MazorX™, Mazor X™ Align, and/or Stealthstation™. In an optional step 114,surgical treatment configuration holographic image 23 is superimposedwith body image 20. In an optional step 116, surgical treatmentconfiguration image 23 and body image 20 are displayed from headset 12.In an optional step 118, surgical treatment configuration image 23 issuperimposed with image 18 of the vertebral tissue. In an optional step120, image 18 of the vertebral tissue, surgical treatment configurationimage 23 and body image 20 are displayed from headset 12.

In some embodiments, system 10 includes a method 200 for spinal disorderdiagnosis and treatment planning, as shown in FIG. 12 , similar tomethod 200, as shown in FIG. 11 . In a step 202, a body includingvertebral tissue in a non-surgical environment is imaged to generateimaging 16. In a step 204, a surface of the body adjacent to thevertebral tissue is scanned in real-time with headset 12. Headset 12includes camera 26 which includes, for example, a depth sensing camera,as described herein. In a step 206, holographic image 18 of thevertebral tissue with body image 20 of the scanned surface in a commoncoordinate system is registered. In a step 208, holographic image 18 ofthe vertebral tissue and body image 20 in the common coordinate systemis displayed in real-time with headset 12. In an optional step 210,surgical treatment configuration 22 for the vertebral tissue isdetermined. In some embodiments, surgical treatment configuration 22includes a segmentation of the vertebral tissue and/or a surgicalreconstruction of the vertebral tissue.

In an optional step 212, surgical treatment configuration image 23 withbody image 20 in the common coordinate system is registered. In anoptional step 214, surgical treatment configuration image 23 and bodyimage 20 in the common coordinate system are displayed with headset 12.In an optional step 216, surgical treatment configuration image 23 withimage 18 of the vertebral tissue and body image 20 in the commoncoordinate system are registered. In an optional step 218, image 18 ofthe vertebral tissue, surgical treatment configuration image 23 and bodyimage 20 in the common coordinate system are displayed with headset 12.

In some embodiments, system 10 includes a method 300 for spinal disorderdiagnosis and treatment planning, as shown in FIG. 13 , similar tomethod 100, as shown in FIG. 11 and method 200, as shown in FIG. 12 . Ina step 302, a body including vertebral tissue in a non-surgicalenvironment is imaged to generate imaging 16. In a step 304, a surfaceof the body adjacent to the vertebral tissue is scanned in real-timewith headset 12. In a step 306, surgical treatment configuration 22 forthe vertebral tissue is determined. In a step 308, image 18 of thevertebral tissue and/or surgical treatment configuration image 23 isregistered with body image 20 of the scanned surface in a commoncoordinate system. In a step 310, image 18 of the vertebral tissueand/or surgical treatment configuration image 23 with body image 20 inthe common coordinate system is displayed in real-time with headset 12.

In some embodiments, an image guidance system, navigation system and/ora robotic guidance system, are employed with system 10, method 100,method 200 and/or method 300 if after diagnosing and treatment planning,a surgical procedure is desired. See, for example, similar surgicalnavigation components and their use as described in U.S. Pat. Nos.6,021,343, 6,725,080, 6,796,988, the entire contents of each of thesereferences being incorporated by reference herein. See, for example,STEALTHSTATION® AXIEM™ Navigation System, sold by Medtronic Navigation,Inc. having a place of business in Louisville, Colo. Exemplary trackingsystems are also disclosed in U.S. Pat. Nos. 8,057,407, 5,913,820,5,592,939, the entire contents of each of these references beingincorporated by reference herein.

It will be understood that various modifications may be made to theembodiments disclosed herein. Therefore, the above description shouldnot be construed as limiting, but merely as exemplification of thevarious embodiments. Those skilled in the art will envision othermodifications within the scope and spirit of the claims appended hereto.

What is claimed is:
 1. A spinal disorder diagnosis and treatmentplanning system comprising: a mixed reality holographic displayincluding at least one processor, at least one camera, at least onesensor, and being configured to acquire data points corresponding to asurface of a body adjacent to vertebral tissue; and a computer databaseconfigured to transmit imaging of the body including the vertebraltissue to the mixed reality holographic display, the mixed realityholographic display being configured to display a first holographicimage of the vertebral tissue superimposed with a body image includingthe surface; wherein the mixed reality holographic display is configuredto transmit the data points to the computer database and the computerdatabase is configured to determine a surgical treatment configurationfor the vertebral tissue.
 2. A system as recited in claim 1, wherein themixed reality holographic display is configured to display the firstholographic image superimposed with the body image in real-time.
 3. Asystem as recited in claim 1, wherein the mixed reality holographicdisplay is configured to dynamically display in real-time theorientation and alignment of the vertebral tissue in a non-surgicalenvironment.
 4. A system as recited in claim 1, wherein the surgicaltreatment configuration includes a segmentation of the vertebral tissueand/or a surgical reconstruction of the vertebral tissue.
 5. A system asrecited in claim 4, wherein the mixed reality holographic display isconfigured to display a second holographic image of the surgicaltreatment configuration superimposed with the body image.
 6. A system asrecited in claim 4, wherein the mixed reality holographic display isconfigured to display the first holographic image, the secondholographic image and the body image.
 7. A system as recited in claim 1,wherein the data points include three dimensional mapping of thesurface.
 8. A system as recited in claim 1, wherein the mixed realityholographic display has at least one sensor including a threedimensional scanner for acquiring the data points and at least oneprocessor including a registration processor for superimposing the firstholographic image with the body image.
 9. A system as recited in claim1, wherein the imaging includes a radiograph.
 10. A system as recited inclaim 8, wherein recognition markers are positioned relative to thesurface to provide anatomic landmarks for the three dimensional scanner.11. A system as recited in claim 1, wherein the mixed realityholographic display includes at least one depth sensing camera.
 12. Asystem as recited in claim 1, wherein the mixed reality display includesan optical see-through headset.
 13. A spinal disorder diagnosis andtreatment planning system comprising: a tangible storage devicecomprising computer-readable instructions; a mixed reality holographicheadset including a central processor and a holographic processor, andone or more cameras and sensors; and one or more processors, executingthe instructions in operation of the system for: imaging a bodyincluding vertebral tissue in a non-surgical environment; scanning inreal-time a surface of the body adjacent to the vertebral tissue withthe mixed reality holographic headset; registering a first holographicimage of the vertebral tissue with a body image of the scanned surfacein a common coordinate system; and displaying in real-time the firstholographic image and the body image in the common coordinate systemwith the mixed reality holographic headset, wherein the one or moreprocessors further determine a surgical treatment configuration for thevertebral tissue.
 14. A system as recited in claim 13, wherein thesurgical treatment configuration includes a segmentation of thevertebral tissue and/or a surgical reconstruction of the vertebraltissue.
 15. A system as recited in claim 13, wherein the mixed realityholographic display is configured to display a second holographic imageof the surgical treatment configuration registered with the body image.16. A system as recited in claim 13, wherein the scanning includes threedimensional mapping of the surface.
 17. A system as recited in claim 13,wherein the mixed reality holographic headset includes at least onedepth sensing camera.
 18. A spinal disorder diagnosis and treatmentplanning system comprising: a tangible storage device comprisingcomputer-readable instructions; a mixed reality holographic headsetincluding a central processor and a holographic processor, and one ormore cameras and sensors; and one or more processors, executing theinstructions in operation of the system for: imaging a body includingvertebral tissue in a non-surgical environment; scanning in real-time asurface of the body adjacent to the vertebral tissue with the mixedreality holographic headset; registering a first holographic image ofthe vertebral tissue with a body image of the scanned surface in acommon coordinate system; displaying in real-time the first holographicimage and the body image in the common coordinate system with the mixedreality holographic headset; and determining a surgical treatmentconfiguration for the vertebral tissue, wherein the surgical treatmentconfiguration includes a segmentation of the vertebral tissue and/or asurgical reconstruction of the vertebral tissue.
 19. A system as recitedin claim 18, wherein the mixed reality holographic display is configuredto display a second holographic image of the surgical treatmentconfiguration registered with the body image.
 20. A system as recited inclaim 18, wherein the scanning includes three dimensional mapping of thesurface.